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Method and Apparatus for Rules-Based Storage and Retrieval of Multimedia 
Interactions Within a Communication Center 

by Inventor 
Christopher Clemmett Macleod Beck et al 

Field of the Invention 



The present invention is in the field of telecommunication encompassing all 
10 existing sorts of interaction multimedia technology, and pertains more particularly to 
methods and apparatus for storing a complete record of multimedia interactions within 
a multimedia communication center using existing enterprise rules. 



15 Cross-reference to related documents 

The present application is a continuation-in-part (ClP)of copending application 
P3314PA, which is a C1P of copending application P3313PA, both of which are 
incorporated herein by reference. 

20 

Background of the Invention 

In the field of telephony communication, there have been many improvements 
25 in technology over the years that have contributed to more efficient use of telephone 
communication within hosted call-center environments. Most of these improvements 
involve integrating the telephones and switching systems in such call centers with 
computer hardware and software adapted for. among other things, better routing of 
telephone calls, faster delivery of telephone calls and associated information, and 
30 improved service with regard to client satisfaction. Such computer-enhanced 
telephony is known in the art as computer-telephony integration (CTI). 
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Generally speaking, CTI implementations of various design and purpose are 
implemented both within individual call-centers and, in some cases, at the telephone 
network level. For example, processors running CTI software applications may be 
linked to telephone switches, service control points (SCPs) : and network entry points 

5 within a public or private telephone network. At the call-center level, CTI-enhanced 
processors, data servers, transaction servers, and the like, are linked to telephone 
switches and, in some cases, to similar CTI hardware at the network level, often by a 
dedicated digital link. CTI processors and other hardware within a call-center is 
commonly referred to as customer premises equipment (CPE). It is the CTI processor 

10 and application software is such centers that provides computer enhancement to a call 
center. 

In a CTI-enhanced call center, telephones at agent stations are connected to a 
central telephony switching apparatus, such as an automatic call distributor (ACD) 
switch or a private branch exchange (PBX). The agent stations may also be equipped 

15 with computer terminals such as personal computer/video display unit's (PC/VDU's) 
so that agents manning such stations may have access to stored data as well as being 
linked to incoming callers by telephone equipment. Such stations may be 
interconnected through the PC/VDUs by a local area network (LAN). One or more 
data or transaction servers may also be connected to the LAN that interconnects agent 

20 stations. The LAN is, in turn, typically connected to the CTI processor, which is 
connected to the call switching apparatus of the call center. 

When a call arrives at a call center, whether or not the call has been pre- 
processed at an SCP, typically at least the telephone number of the calling line is 
made available to the receiving switch at the call center by the network provider. This 

25 service is available by most networks as caller-ID information in one of several 

formats such as Automatic Number Identification (ANI). Typically the number called 
is also available through a service such as Dialed Number Identification Service 
(DNIS). If the call center is computer-enhanced (CTI), the phone number of the 
calling party may be used as a key to access additional information from a customer 

30 information system (CIS) database at a server on the network that connects the agent 
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workstations. In this manner information pertinent to a call may be provided to an 
agent, often as a screen pop on the agent's PC/VDU. 

In recent years, advances in computer technology, telephony equipment, and 
infrastructure have provided many opportunities for improving telephone service in 

5 publicly-switched and private telephone intelligent networks. Similarly, development of a 
separate information and data network known as the Internet, together with advances in 
computer hardware and software have led to a new multimedia telephone system known in 
the art by several names. In this new systemology, telephone calls are simulated by 
multimedia computer equipment, and data, such as audio data, is transmitted over data 

10 networks as data packets. In this system the broad term used to describe such computer- 
simulated telephony is Data Network Telephony (DNT). 

For purposes of nomenclature and definition, the inventors wish to distinguish 
clearly between what might be called conventional telephony, which is the telephone service 
enjoyed by nearly all citizens through local telephone companies and several long-distance 

15 telephone network providers, and what has been described herein as computer-simulated 
telephony or data-network telephony. The conventional systems are referred to herein as 
Connection-Oriented Switched-Telephony (COST) systems, CT1 enhanced or not. 

The computer-simulated, or DNT systems are familiar to those who use and 
understand computers and data-network systems. Perhaps the best example of DNT is 

20 telephone service provided over the Internet, which will be referred to herein as Internet 
Protocol Network Telephony (IPNT), by far the most extensive, but still a subset of DNT. 

Both systems use signals transmitted over network links. In fact, connection to data 
networks for DNT such as 1PNT is typically accomplished over local telephone lines, used 
to reach points in the network such as an Internet Service Provider (ISP). The definitive 

25 difference is that COST telephony may be considered to be connection-oriented telephony. 
In the COST system, calls are placed and connected by a specific dedicated path, and the 
connection path is maintained over the time of the call. Bandwidth is basically assured. 
Other calls and data do not share a connected channel path in a COST system. A DNT 
system, on the other hand, is not dedicated or connection-oriented. That is. data, including 

30 audio data, is prepared, sent, and received as data packets over a data-network. The data 
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packets share network links, and may travel by varied and variable paths. 

Recent improvements to available technologies associated with the 
transmission and reception of data packets during real-time DNT communication have 
enabled companies to successfully add DNT, principally IPNT, capabilities to existing 

5 CTI call centers. Such improvements, as described herein and known to the inventor, _ 
include methods for guaranteeing available bandwidth or quality of service (QoS) for 
a transaction, improved mechanisms for organizing, coding, compressing, and 
carrying data more efficiently using less bandwidth, and methods and apparatus for 
intelligently replacing lost data via using voice supplementation methods and 

10 enhanced buffering capabilities. 

In addition to Internet protocol (IPNT) calls, a DNT center may also share 
other forms of media with customers accessing the system through their computers. 
E-mails, Video mails, fax, file share, file transfer, video calls, and so forth are some of 
the other forms of media which may be used. This capability of handling varied 

15 media leads to the term multimedia communications center. A multimedia 

communications center may be a combination CTI and DNT center, or may be a DNT 
center capable of receiving COST calls and converting them to a digital DNT format. 
The term communication center will replace the term call center hereinafter in this 
specification when referring to multimedia capabilities. 

20 In typical communication centers, DNT is accomplished by Internet 

connection and IPNT calls. For this reason, 1PNT and the Internet will be used in 
examples to follow. IT should be understood, however, that this usage is exemplary, 
and not limiting. 

In systems known to the inventors, incoming IPNT calls are processed and 
25 routed within an IPNT-capable communication center in much the same way as COST 
calls are routed in a CTI-enhanced call-center, using similar or identical routing rules, 
waiting queues, and so on, aside from the fact that there are two separate networks 
involved. Communication centers having both CTI and IPNT capability utilize LAN- 
connected agent-stations with each station having a telephony-switch-connected 
30 headset or phone, and a PC connected, in most cases via LAN, to the network carrying 
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the IPNT calls. Therefore, in most cases, IPNT calls are routed to the agent's PC 
while conventional telephony calls are routed to the agent's conventional telephone or 
headset. Typically separate lines and equipment must be implemented for each type 
of call weather COST or IPNT. 

Due in part to added costs associated with additional equipment, lines, and 
data ports that are needed to add IPNT capability to a CTl-enhanced call-center, 
companies are currently experimenting with various forms of integration between the 
older COST system and the newer IPNT system. For example, by enhancing data 
servers, interactive voice response units (lVR's), agent-connecting networks, and so 
on, with the capability of conforming to Internet protocol, call data arriving from 
either network may be integrated requiring less equipment and lines to facilitate 
processing, storage, and transfer of data. 

With many new communication products supporting various media types 
available to businesses and customers, a communication center must add significant 
application software to accommodate the diversity. For example, e-mail programs 
have differing parameters than do IP applications. IP applications are different 
regarding protocol than COST calls, and so on. Separate routing systems and/or 
software components are needed for routing e-mails, IP calls, COST calls, file 
sharing, etc. Agents must then be trained in the use of a variety of applications 
supporting the different types of media. 

Keeping contact histories, reporting statistics, creating routing rules and the 
like becomes more complex as newer types of media are added to communication 
center capability. Additional hardware implementations such as servers, processors, 
etc. are generally required to aid full multimedia communication and reporting. 
Therefore, it is desirable that interactions of all multimedia sorts be analyzed, 
recorded, and routed according to enterprise (business) rules in a manner that provides 
seamless integration between media types and application types, thereby allowing 
agents to respond intelligently and efficiently to customer queries and problems. 

One issue that continually plagues current art communication centers is an 
inability of agents to keep track of customer interactions in an orderly fashion. For 
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example, if a certain customer has a service problem that he has contacted the center 
about, it is desired that all past queries and responses regarding the issue be known to 
an agent so that much information is not rehashed or may be even missed completely. 
Some current art systems known to the inventor support customer history 

5 systems wherein a chronological interaction history regarding a certain customer is 
available via a database to an agent or agents of the communication center. This 
function is typically performed via threading the queries and replies in a chronological 
fashion wherein the customer has an identifying string. However, current art systems 
support only a few types of text media such as e-mail, fax or other text-type 

10 documents. If an agent places a IP call to the customer, or perhaps, enlists another 
agent in dealing with the customer using an alternate media, manual intervention must 
be used in order to update the customer's transaction history. This presents a problem 
in that the agent or agents may forget to enter certain information into the system. In 
the event of an audit, the customers string would be incomplete and would not 

15 accurately reflect actual history and status. 

What is clearly needed is a multimedia threading technique that automatically 
enters information to the customer interaction history as it happens regardless of 
media type, and also provides a method for retrieval and auditing. 

20 

Summary of the Invention 

In a preferred embodiment of the present invention a system for relating 
multimedia events in a multimedia-capable call center, and making the events 

25 available to users is provided, comprising a data repository adapted to store files 
recording both text-based and non-text-based events involving the call center; 
recording systems for recording events of all text-based and non-text events 
experienced at the call center; a system for relating the stored files in one or more 
serial strings according to relational criteria; and an interactive display interface for 

30 displaying identifiers of the stored files, the identifiers arranged in a serial string 
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according to the relational criteria. Selected events involving the call center are 
recorded, and a user, be selecting identifiers, may play individual ones of the stored 
files, thereby reviewing the recorded events. 

In this system the identifiers may comprise icons displayed on a computer 

5 video monitor, the icons arranged in a serial string. There may further be a system for_ 
preparing text versions of non-text events, and for storing the text versions in the data 
repository related to the non-text versions in a manner that the text versions may be 
accessed by the interactive display interface. There may still further be a data 
extraction facility for deriving information from text files, including the text versions 

10 of non-text files. Still further a facility may be provided for relating informational 
files to text files, and therefore to non-text files associated with text files, wherein the 
informational files may also be accessed by the interactive display interface along 
with the recorded events and text versions of recorded events. 

Relational criteria may include one or more of chronology, participants in an 

15 event, organizational affiliation, and single issue relationship, but are not limited to 
these criteria. In some cases all communication events involving the call center are 
copied, related and stored, and in some cases a text version of every non-text event is 
prepared, related and stored. Events may include one or more of e-mails, video-mails, 
faxes, file share, file transfers, video calls, chat sessions, DNT calls, COST calls, and 

20 voice mails. 

In another aspect of the invention a method is provided for relating multimedia 
events in a multimedia-capable call center, and making the events available to users, 
comprising steps of (a) storing both text-based and non-text-based events involving 
the call center in a data repository adapted to store multimedia files; (b) relating the 
25 stored files in one or more serial strings according to relational criteria; and (c) 

providing an interactive display interface on a computer video monitor, the interface 
adapted for displaying identifiers of the stored files, the identifiers arranged in a serial 
string according to the relational criteria, wherein user, be selecting identifiers, may 
play individual ones of the stored files, thereby reviewing the recorded events. 
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In the method, in step (c), the identifiers may comprise icons displayed on the 
computer video monitor, the icons arranged in a serial string. There may further be a 
step for preparing text versions of non-text events, and for storing the text versions in 
the data repository related to the non-text versions in a mariner that the text versions 

5 may be accessed by the interactive display interface. There may still further be a step 
for extracting information from text files, including the text versions of non-text files. 
There may be yet another step for relating informational files to text files, and 
therefore to non-text files associated with text files, wherein the informational files 
may also be accessed by the interactive display interface along with the recorded 

1 0 events and text versions of recorded events. 

In the method the relational criteria may include one or more of chronology, 
participants in an event, organizational affiliation, and single issue relationship, 
although there may well be other relational criteria. In some cases, in step (a), all 
communication events involving the call center are copied, related and stored. Text 

15 versions of every non-text event may be prepared, related and stored. Also, events 
may include one or more of e-mails, video-mails, faxes ; file share, file transfers, video 
calls, chat sessions, DNT calls, COST calls, and voice mails. 

In the multimedia call center system and methods taught in enabling detail 
below, functions are provided for use by clients of and workers at a call center never 

20 before available in the art, providing a multimedia call center of broad use and 
effectiveness. 



25 



Brief Description of the Drawing Figures 

Fig. 1 is a diagram of a multimedia communications center enhanced with a 
network operating system according to an embodiment of the present invention. 

Fig. 2 is a block diagram illustrating basic layers of a customer interaction 
operating system according to an embodiment of the present invention. 
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Fig. 3 is a flow chart illustrating basic steps performed by the network 
operating system of Fig. 2 related to completing interactive transactions between 
business partners. 

Fig. 4 is a block diagram illustrating agent-desktop function according to an 
5 embodiment of the present invention. 

Fig. 5 is a block diagram of an exemplary WEB-form customer interface 
according to an embodiment of the present invention. 

Fig. 6 is a flow chart illustrating media-presentation and customer-interface 
logic steps according to an embodiment of the present invention. 
10 Fig. 7 is an exemplary overview of a multimedia interaction storage system 

within a communication center according to an embodiment of the present invention. 

Fig. 8 is a block diagram of the repository of Fig. 7 illustrating threaded text- 
blocks and their relationship to stored multimedia according to an embodiment of the 
present invention. 

15 Fig. 9 is a process flow chart illustrating logical steps taken when building a 

threaded multimedia contact-history of communication-center interactions according 
to an embodiment of the present invention. 



20 Description of the Preferred Embodiments 

Fig. 1 is a multimedia communications center enhanced with a network 
operating system according to an embodiment of the present invention. A telephony- 
network architecture 1 1 comprises an enterprise-hosted communication center 17 that 
25 is linked to ? in this example, both a publicly-switched telephone network (PSTN) 13 ? 
and a wide area network (WAN) 15, which may be the public Internet or other digital 
network, such as a company Intranet. 

In this particular embodiment communication center 1 7 handles both 
conventional telephone calls, which may be categorized as connection oriented 
30 switched telephony (COS T) calls, and data network telephony (DNT) calls, which 
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may be DNT calls over a private digital network or calls according to a protocol such 
as the well-known Internet protocol. DNT calls are characterized in that data is 
transmitted as addressed data packets as opposed to dedicated connections in COST 
calls. As indicated, PSTN 13 may be a private rather than a public network. WAN 1 5 

5 may be a company Intranet, the Internet, or another type of WAN known in the art. 
The particular method of call deliver}' and call center integration is not particularly 
relevant for the purposes of this invention. There are many ways known both to the 
inventor as well as known in the art. Particular issues discussed in the disclosure 
between the telephones and the computers might be implemented differently 

10 depending on the actual system, but shall be deemed equivalent for all purposes of 
this invention. 

Incoming COST calls arrive at a network-level telephony switching apparatus 
19 in network cloud 13 and are connected over trunk 23 to a central telephony 
switching apparatus 27 within communication center 17. From switching apparatus 
15 27, calls are routed according to existing routing rules over interna] wiring 56 to 

agents' telephones 47, 49, 51, and 53 residing at agents' workstations 31, 33, 35, and 
37 respectively. 

Incoming DNT calls, and other communication events such as e-mail, file 
transfers and the like, arrive at a routing node 21 in WAN 15 and are passed on over 

20 digital connection 25 to a routing server 29 within communication center 1 7. Once 
calls arrive at server 29, they may, in some embodiments, be routed directly over 
LAN 55 according to existing routing rules to personal computer/video display units 
(PC/VDU) such as PC/VDU 39, 41 , 43, or 45 located at agent's workstations 31, 33, 
35, and 37 respectively. 

25 In this embodiment, switch-connected telephones 47-53 are also connected to 

PC/VDU *s 39-45 via a headset to computer sound-card according to technique known 
to the inventor and accomplished via an I/O cable. Thus connected, agents may 
respond to incoming COST and DNT calls with the same headset. 

In the exemplary system and communication center shown, the equipment and 

30 applications are adapted to provide for multimedia operation at each of the agent 
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stations, so the agents can interact with clients in many different ways, as are known 
in the multimedia arts. 

Computer telephony integration (CTJ) enhancement is, in this embodiment, 
provided both at communication center 1 7 and in PSTN 1 3. For example, in PSTN 
13, a processor 61 running instances of a CT1 application known as a T-server (TS) to__ 
the inventors, and a statistics server (Stat) is connected to telephony switch 19 via CT1 
link 65. An intelligent peripheral 59 of the form of an interactive voice response unit 
(1VR) is connected to processor 61 via data connection 63. Similar CTI equipment is 
illustrated within communication center 17. Namely, a processor 67 running 
instances of TS and Stat and connected to telephony switch 27 via CTI link 71, and an 
IVR 69 connected to processor 67 via a data connection 73, with processor 67 further 
connected to a local area network (LAN) 55 within communication center 17. 

In alternative embodiments there may also be a CTI processor 22 in WAN 15 
connected to server 2 1 by a CTI link 24. Also in some embodiments a separate data 
network 66 connects these CTI processors. In this way, intelligent routing may be 
performed at the network level with negotiation and direction from within 
communication center 17. 

It will be appreciated by those with skill in the art that the CTI enhancements, 
as immediately described above, may be hosted on one processor at PSTN 1 3 and on 
one processor at communication center 1 7 without departing from the spirit and scope 
of the present invention. The inventor has chosen to show separate processors having 
separate functions for exemplary purposes only. It will also be appreciated by the 
skilled artisan that there may be many more or fewer than the four agent stations 
shown in communications center 1 7, and hardware and software arrangements may be 
made is a variety of ways. Also, home agents might be connected in a variety of ways 
to the call center. 

In a preferred embodiment of the present invention, a customer-interaction 
network operating system, hereinafter termed (CINOS), is provided for the purpose of 
managing communications center 17, and optimizing and recording all agent/customer 
interactions received at communication center 17 from networks 13 and 15. CINOS is 
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unique in the fact that it is a multi-tiered object-and process-orientated system wherein 
logic regarding the various aspects of it's functionality is achieved via knowledge- 
based architecture and object modeling. Various functions of CINOS, more fully 
described below, include capturing (recording), analyzing, routing, and, in many 

5 instances, responding via automated process to customers engaged in interactions with^ 
the enterprise (company hosting the communication center). CINOS is adapted to 
support all planned communication mediums such as multimedia DNT applications 
including e-mail, video mail, file transfers, chat sessions, IP calls, and CT1 COST 
transactions such as voice calls, voice mails, faxes, and so on. 

10 Referring back to Fig. 1 , CINOS utilizes various LAN-connected machines in 

order to perform various operations. Among these various hardware implementations 
are a multimedia server (MIS) 79 adapted to physically store and serve all multimedia 
transactions, and a customer-information-system server (CIS) 57 adapted to physically 
store and serve information relevant to customers such as purchase history, financial 

15 status, product preferences, contact information, etc. A central server (COS) 77 acts 
as a host location for a CINOS manager application (noted in text balloon) which is, 
in effect, the parent application that controls all of the operation and functionality of 
the system. 

In addition to the above-mentioned machines hosting CINOS routines, each 
20 PC/VDU such as PC/VDU 39, for example, has a CINOS-agent desktop interface or 
client application (not shown) adapted to interact with the parent application. Also, 
each machine that provides particular dedicated function to communication center 17 
such as switch-connected CT1 processors, IVR's, and other related equipment host 
instances of CINOS application-program interfaces (API's) to enable seamless 
25 integration of differing parameters and/or protocols that are used with various planned 
application and media types utilized within communication center 17. Such programs 
may also co-reside or be in any combination or hosted by themselves. Additionally, 
for performance purposes, additional dedicated network links may exist between those 
servers, but essentially they are only performance boosters, and hence for clarity 
30 purposes, only a simple network is shown. 
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As previously described, CINOS comprises a multi-tiered architecture. This 
unique architecture comprises an external media layer for interfacing with the 
customer or business contact, a workflow layer for making routing decisions, 
organizing automated responses, recording transactions, and so on, and an internal 
media later for interfacing and presenting interactions to an agent or knowledge 
worker. An innovative concept associated with CINOS involves the use of tooled 
process models, knowledge bases, and other object models as base instruction for it's 
various functions. These modular conventions may be inter-bound with each other, 
and are easily editable providing a customizable framework that may conform to 
virtually any existing business logic. 

In simple operation, and after any network level routing, COST calls and DNT 
calls including other media events arrive at communication center 17 to telephony 
switch 27, and routing server 29 respectively. Network level routing, as defined 
herein, includes any intelligent implementation that may be in place and aided via 
processors 59, 61, and 22. Load balancing to multiple communication centers, and 
transferring customer data obtained at network-level over data-network connection 66 
would be examples of such network-level routing. 

Once a call or other communication event registers at either switch 27 or 
routing server 29, CINOS immediately identifies the media type associated with the 
call and begins it's processes depending on enterprise rules. For example, a live 
COST call may first be routed to IVR 69 whereby the customer can be presented with 
varying choices such as leaving a voice message, waiting in queue, receiving a call 
back, or perhaps an e-mail, and so on. Interaction by IVR 69, in this instance, will 
preferably be via voice recognition technique such as is known in the art, but may also 
be via touch tone response or other known method. As previously described, the 
caller may elect from a number of options, such as to hold for a next available agent, 
select an automated response such as a fax back, or perhaps a later agent-initiated 
response such as an e-mail or call back. In all cases, CINOS seamlessly processes and 
executes the logic required to accomplish the goal of the caller in a media and 
application-independent fashion. 
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DNT events are handled much the same way as described above for live 
callers. For example, an IP call may be routed to a digital equivalent of an 1VR for 
interaction or queued for a next available agent, and so on. In one embodiment, IVR 
69 may be adapted to handle both COST and DNT interaction. 

All interactions with live external media, including actual text-based events 
whether live or not, are recorded and stored in MIS 79 with an associated text version 
of the media stored as well, and becoming part of an overall threaded contact history. 
This is accomplished in varying ways according to existing parameters such as media 
type, whether the event is a live call, and so on. For example, CINOS may execute a 
command directing IVR 69 to digitally record an incoming COST call during 
customer interaction and then store the voice recording of the transaction in MIS 79. 
A text version of the recording either created simultaneously from the voice recording 
via voice-to-text techniques (known in the art), or created by a live attendant via 
manual annotation may be sent to and stored in DB 79. An IPNT call arriving at 
routing server 29 may be similarly recorded and stored in MIS 79 with an associated 
text version of the interaction stored in DB 79. E-mails, video calls, voice mails and 
so on are similarly handled. For example, an incoming e-mail is stored in MIS server 
79 while text from the e-mail may be extracted and stored associated with the e-mail. 

The purpose of the text version of the event is twofold. Firstly, a complete 
text-based transaction history of communication center 17 may be compiled and 
reserved for later access and audit. Secondly, an agent or knowledge worker may, in 
some instances, see the text version of the event at the same time that he receives 
routed notification of the event. In this way, an agent may begin mental preparation 
before taking a call. The text version of an event must be machine-readable and 
human readable at times displayed. Interactive media-independent viewers, part of 
the agent's client application, may be used to disseminate information which may 
initially not be human readable. 

It is important to note here that the text-based version of an event may or may 
not be a complete and verbatim rendition of an actual media event. For example, an 
e-mail may contain many documents each having many pages of text. Therefore, the 
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text-based version of a particular e-mail event may simply contain the name and 
particulars regarding the author, a purchase order, and a list of the enclosed documents 
by title, and basic content or memo as well as a possible manual annotation. The 
attachments to the e-mail may be stored separately, and be also cross-indexed and 

5 retrievable. Seeing the purchase order when the event is routed to an agent desktop 
tells the agent that this e-mail is important. 

A fax, stored originally as a bit-mapped document, may be converted to text in 
the system via optical recognition (OCR) technique wherein sometimes only certain 
content such as the authors contact information, basic intent of the fax, and perhaps 

]0 special numbers or codes contained in the original fax are recorded in a text version 
79 , sometimes the whole text is OCR'd, while the original fax is stored in it's entirety 
in DB 79. Such codes or numbers that are specifically parsed from actual media may 
be part of a unique coding system set up by the enterprise whereby customers are 
directed to include such codes or numbers with their orders, service requests, and so 

15 on. 

Parsing text messages is accomplished via a text-analyzer known to the 
inventor. In other non-text media types, such as video or graphics, descriptive notes 
may be taken via live attendant and stored in DB 79 as previously mentioned. Voice 
recognition technology may also be used in a case of recorded sound or video with 

20 sound. All transactions regardless of media type are thus recorded and stored 

according to enterprise rules with at least a meaningful part of the content if not all of 
the content of such transactions converted to text and stored in DB 79 associated with 
the recording of the event. Again, the importance of the text version is that the 
extracted knowledge of the transaction therein is in machine-operable code, allowing 

25 search and cross-referencing functions that may otherwise not be possible. 

After incoming events are analyzed and processed with regards to queuing, 
recording, storing, etc. CINOS decides the disposition paths of each event. For 
example, live calls in queue are routed to live agents if available, if this is the priority 
action in the enterprise rules, fc-mails are either routed to next available agents using 

30 a push technology, or simply stored in MIS server 79 where they may be retrieved by 
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agents after receiving notification. Recorded events such as I VR voice requests are 
stored in MIS server 79 where they may be retrieved by agents, and so on. 

By the use of routing and routing notification events, any media may be routed 
to an appropriate agent based on skill, or any other rule-based routing method over 
5 LAN 55. Actual multimedia events may be accessed from MIS server 79 at the 

agent's discretion, or by rule, and text-based versions of those events stored in DB 79 
may be mirrored and routed to the agent along with notification of the incoming 
event. 

Other services may be performed by C1NOS such as responding to media 

10 requests without agent participation via initiating automated fax responses, out-bound 
dialing campaigns wherein recorded information is given to a customer perhaps 
concerning an order placed by the customer, and so on. Networking via business or 
chat applications between several business partners, customers, agents, and so on, is 
possible wherein each entry may be stored in DB 79 as part of a discussion thread 

15 including responses of another media type, perhaps initiated by a communication- 
center agent to one of the participants during the discussion. 

As a general rule, full multimedia storage is done in a mass storage server, and 
linked by cross-indexing to the database. Depending on the business model, full text 
or only partial annotation is stored in the database, or a mix therof, e.g by media type. 

20 In addition to supporting a wide variety of applications and protocol, C1NOS 

is provided with the tools for building media-independent self-help wizards that are 
adapted for problem solving and reduction. Similarly, external and internal 
interaction media viewers are provided and adapted to support any media of choice. 
C1NOS uses object modeling and linking techniques that are known in the art 

25 to effect much of it's goal of presenting a seamless customer interaction with an 
enterprise agent or knowledge worker operating in a communication center such as 
center 1 7. For example, an interaction object model (IOM) represents a transcript of 
all interaction history stored in DB 79 and provides an audit trail of the state of 
transactions of all interactions. An interaction process model (IPM) controls how 

30 events are handled within the operating system. 
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An additional set of models handle how agents receive their routed media such 
as via traditional push model, blended push model, publish and subscribe model, or 
interrupt model. Prioritizing interaction events may also be accomplished through 
varying the push theme or scheme. For example, traditional push technology for e- 

5 mail means that only e-mail (media type) is being worked on by an agent. By 
blending the push model with a publish and subscribe model, the interrupt model is 
created wherein the agent may subscribe to various routed media such as answering 
phones, and responding to faxes, but may be interrupted for an important interaction 
of another media type such as e-mail and so on. In this way an agent's time may be 

10 utilized according to enterprise rules within an automated environment. 

Outbound campaigns may be configured according to enterprise rules and 
media preference using a single rule-set knowledge-base. This single set of outbound 
tools can be used to initiate customer dialog via predictive dialing, e-mail push, 
automated recorded messages, and so on. 

15 It will be apparent to those with skill in the art that common object modeling 

(COM) can be used to create virtually any type of model for any type of enterprise 
situation. It is the intention of the inventor to provide the applicable control codes 
known in the art for building process and object models and enabling the linking and 
interaction between the models. As previously described, it is partly the fact that 

20 CINOS uses these various models and knowledge bases to achieve desired interaction 
that sets it above current-art systems. The inventor knows of no such network 
interfacing operating system that is based on the above described technology. 

CINOS may be implemented in a number of different topologies. For 
example, CINOS may be implemented as a centralized topology with one 

25 communication center as shown here in Fig. 1 , a distributed topology wherein a single 
communication center may span multiple physical locations, a segmented 
communication center wherein a single pool of agents services more than one 
company or customer base, or a wide communication network wherein a plurality of 
communication centers such as center 1 7 cooperatively service a common pool of 
30 customers or a customer base. Enterprises involved in commerce such as large 
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fmancial institutions hosting many geographically separate communication centers 
may build their entire networking system using CINOS architecture in standardized 
and distributed fashion. There is no limitation to the type of enterprise that may use 
CINOS as it may be tooled to accommodate virtually any network architecture linked 

5 to a communication center having DNT capability. 

it will also be apparent to one with skill in the art that CINOS routines 
according to various embodiments of the present invention may be included and 
implemented at the network level without departing from the spirit and scope of the 
present invention such as in processor 61, and IVR 59 in PSTN 13, or in routing node 

10 21 in WAN 11. 

Fig. 2 is a block diagram illustrating basic layers of the network operating 
system according to an embodiment of the present invention. As previously described 
with reference to Fig. 1 , CINOS comprises three basic operating layers. They are an 
external media layer 83, a workflow layer 85 ? and an internal media layer 87. 

! 5 External media layer 83 interfaces directly with the customers or business contacts or 
partners as illustrated via customers a and b, and business contact c. The bi- 
directional arrows beneath each of the above mentioned participants illustrate 
interactive participation with CINOS on the customer side. 

External media layer 83 may. in one embodiment, be a multifaceted, web- 

20 based self-help interface providing news information and a host of other services that 
may be personalized by the customer. In many respects, external media layer 83 in 
this embodiment is similar to a web browser. 

Workflow layer 85 comprises 3 basic function categories beginning with a 
content analysis category 89 wherein textual analysis, voice analysis, I VR interaction, 

25 recording and storing takes place. A next category is context resolution 91 . Context 
resolution involves customer identification, business process binding, preparation for 
routing, and so on. A third category termed interaction routing 93 comprises various 
processes associated with the presentation of the interaction to agents, service persons, 
knowledge workers, business partners, customers and the like, that is, all transaction 
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partners. Category 93 covers queuing, skill-based routing, automated treatment, 
workflow models, and so on. 

Internal media layer 87 comprises an agent desktop interface not shown in Fig. 
1, but described in more detail below. Both external layer 83 and internal layer 87 

5 contain the required tools for enabling media and application-independent interfacing _ 
such as previously mentioned self-help wizards, media viewers, and other controls as 
prescribed via enterprise rules. 

Internal media layer 87 provides an agent with, among other options, 
information about the customer or contact, information about current or historical 

10 business processes, information about current interactions and their relationship to 
business processes, and a knowledge-base to guide the agent or knowledge worker 
with interaction response and workflow. An agent a, and agent b, and a knowledge 
worker c are shown herein interacting with the system as illustrated via bi-directional 
arrows. The skilled artisan will recognize these arc merely examples, and there may 

1 5 be many more such persons, and interactions in some instances may be routed to 
machines for response. 

It will be apparent to one with skill in the art that the multi-tiered architecture 
of CINOS such as is illustrated herein may comprise many more or differing steps or 
processes without departing from the spirit and scope of the present invention. 

20 Fig. 3 is a flow chart illustrating basic steps performed by the interaction 

operating system of Fig. 2 related to completing a transaction between a customer and 
an agent wherein the transaction is initiated by the customer. Similar steps may be 
accomplished in the opposite direction for communications initiated by an agent, as 
the system is bi-directional, but the present example will serve to teach the inventive 

25 aspects of the system. In step 95, an incoming transaction, such as a live call, an e- 
mail, etc. , is received at the appropriate CT1 switch (COST) or routing server (DNT) 
in a CINOS communication center such as center 1 7. In step 97, customer and media 
type are identified and interaction proceeds. 

All transactions, whether live calls, such as video calls, DNT calls and COST 

30 calls, or text-based documents, such as e-mails, are recorded and stored in one or more 
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mass storage devices handled by one or more database applications. This may be 
taken as server 79 of Fig. 1 , although the diagram of Fig. 1 is exemplary. 

A principle object of the invention is to extract maximum information from 
every transaction for building a knowledge base that can be used for dynamic 
5 management and future analysis and development. This is done primarily by data 
mining, which is applicable to machine-operable code, that is text. Because of the 
nature of the extraction, there is a difference in the way live calls and text-based 
media is handled. 

Discrimination as to the text nature of the media is made at step 99. If the 

10 media chosen by the customer is already text-based, then the transaction is recorded as 
received (101), and a data mining application extracts important information in step 
103 and stores it in the knowledge base. The distinct portions and versions of the 
transaction, such as the originally recorded version and any extracted data are related 
to one another and to other knowledge previously stored, and become part of a 

15 threaded interaction history associated with an ongoing interaction and ultimately of 
an overall contact history. 

If the media chosen by the customer is determined in step 99 to be a live 
interaction such as a COST or 1PNT call, then the existing knowledge base is accessed 
at step 1 07. and the call is routed to the best fit agent. This may, of course, be done in 

20 a number of ways, such as an ADC, skill-based routing as known to the inventors, 
transfer to an IVR for automatic processing, and so on, as may be dictated by 
enterprise rules. If routing is to an agent, customer information may be retrieved from 
CIS server 57 (Fig. 1) and sent to the agent's PC, and appropriate scripts may be 
provided to guide an agent in interacting with the caller. 

25 In step 1 09 the actual transaction is recorded as it takes place, which, in the 

case of live calls, may be a video or an audio recording or a combination of both. 
Preferably the recording is digitized. 

In step 1 1 1. a maximal text version is prepared from the actual transaction. 
The ability to do so depends to a degree on the sophistication of the system. This 
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process may be as simple as a person adding notes for annotation or as sophisticated 
as a voice-to-text application preparing a full text version as the transaction transpires. 

In step 1 13 the text version is mined for data and resulting knowledge is stored 
in the appropriate knowledge base for future use, and added to overall record with 

5 appropriate cross-referencing. 

It will be apparent to one with skill in the art that there will be many routines 
comprising various steps for performing different processes as may be determined by 
enterprise rules which may likewise vary depending on, among other considerations, 
company type, product and or service type, communication center architecture. 

10 whether or not the system architecture is centralized or distributed, and so on. The 
embodiment taught herein is meant only as a basic example of process functionality 
related to C1NOS processing of an incoming event. 

Fig. 4 is a block diagram illustrating agent-desktop function according to an 
embodiment of the present invention. An agent-desktop client 1 15, part of the CINOS 

1 5 overall architecture, enables an agent or knowledge worker to configure and control 
his or her interface to the rest of the system and to external media. Client 1 1 5 may be 
personalized according to a particular agents parameters. A desktop interface 117 
may appear and function much like a personalized web-browser containing many 
similar attributes related to network capabilities including full multimedia function, 

20 software tool kits, linking and embedding capability, and so on. 

An HTML client application 1 19 oversees all of the network capability 
previously mentioned. In this embodiment for example, HTML client 119 
communicates with an Internet information server 121 using HTTP protocol which is 
standard. Client 1 19, if provided minimally, may be used in conjunction with an 

25 Internet browser for full multimedia function. In some embodiments, it may be 

maximally provided to be a fully featured client with full web browser function. For 
example, an agent may create and edit web forms, web pages, embed controls into 
such web-based forms or pages to provide certain customer interaction mechanisms in 
addition to having a fully functional navigation tool at his disposal. 
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In another embodiment. Server 121 may be a server on a private network or 
corporate WAN instead of an Internet server. In a preferred embodiment, however, 
any number of servers on the Internet and/or linked to a WAN other than the Internet 
may communicate with client 1 19 as it intended to support all existing and known 

5 communication protocols. 

A windows client 123 is provided to seamlessly integrate existing applications 
on the agent's PC to network applications and processes. This may be implemented 
via a desktop tool-kit 125 that contains all of the required controls for building, 
integrating and customizing the interface. 

10 A business-logic layer comprises business object models 129, hereinafter 

termed business objects 129. representing contacts, interactions, knowledge-bases, 
events, routing processes, and other system routines. Integration and interaction of 
the various described desktop components with these logics is accomplished via 
common object modeling (COM) which is known in the art and available to the 

15 inventor. Desktop to CTI integration is accomplished via controls provided or created 
with a CTI set of tools or tool kit (not shown). For example, if the enterprise desires 
to blend voice and e-maik the CTI tool kit would be used to build and integrate the 
interface. 

Existing network applications such as CIS, enterprise resource planning 
20 (ERP), Commerce, and the like interact with various business objects using COM and 
may also interact with a physical database using ODBC and SQL. 

Customer Interface Media Window 

25 

According to a preferred embodiment of the present invention, C1NOS access 
by customers of an enhanced multimedia communication center, such as center 17 of 
Fig. 1 , is controlled by means of a customer-facing media interface, by which 
customers may be identified and even categorized according to numerous criteria. In 
30 some cases access may be controlled through subscription, or according to other 
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qualifying criteria such as may be deemed appropriate by the enterprise. For example, 
if the enterprise is an exclusive investment club, membership may be required. 
Categorizing criteria may include demographic information such as income level, 
credit history, or any other attribute that may be quantified and used to categorize a 
5 customer. 

An enterprise-controlled access point may be defined as an interfacing window 
or portal created and maintained at a typical customer entry point in a network as may 
be known in the art. Such interfaces may take the form of a WEB-based customer 
interface (a WEB page), an interactive voice response (1VR) unit, a service control 
10 point (SCP), or some other customer-facing system or apparatus as may be known in 
the art. 

For the purposes of this specification, an example of an enterprise-controlled 
WEB-form access and interface window is illustrated as an example for a preferred 
embodiment- The inventor deems such an interface to be most adept in offering best- 
15 fit media options while remaining accessible to a large customer or client base. 

Fig. 5 is a block diagram of a WEB-form customer interface according to an 
embodiment of the present invention. WEB form 133, hereinafter termed access 
window 133, is provided to be a part of an enterprise's WEB page which may be 
accessed through Internet connection and navigation, as is knowTi in the art. Widow 
20 1 33 is part of the C1NOS software architecture described above, and represents the 
initiation of any customer interaction with the hosting enterprise. A WEB counter 
143 is provided and records the number of visits to window 133. 

Window 133 is built and edited using COM codes available to the inventor 
and typically found in tool kits adapted for the purpose of creating interactive displays 
25 on a WEB page. Such a tool kit may be located on an agent's desktop, perhaps part of 
an agent's HTML client such as client 1 1 9 of Fig. 4. In one embodiment, it may be 
part of a system administrator's tool kit. 

Window 133 contains interactive options directed at various categories and 
functions. For example, a new client section 135 contains interactive options related 
30 to adding a new client to the active customer base of the enterprise. A customer 
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scrvice section 137 contains interactive options presented to existing clients needing 
service. A new order section 1 39 contains interactive options presented to existing 
clients wishing to do new business. 

Each offered interactive option is an embedded executable function having the 

5 appropriate links to other system areas of CINOS such as may be relevant to the 

immediate interaction such as to services offered, routing routines, database searching, 
interaction recording, and so on. 

An innovative function of window 133 is to provide front-end control of 
access to the enterprise by existing and potential clients/customers. For example, as a 

10 client, contact, or potential client interacts with the various media and functional 
options presented by the enterprise in window 1 33, he or she is being directed 
according to enterprise rules in such a way that he or she may first be qualified or not 
to patronize the enterprise. Secondly, the contacting person may be categorized and 
sorted as to type of qualified customer. Thirdly, the person contacting person may be 

15 directed to pre-selected media options by the enterprise for various services offered 
including but not limited to routing live interactions, and so on. 

In a preferred embodiment of the present invention, access window 133 is 
fully customizable, based on customer data and enterprise rules with the focus of such 
customization directed toward benefiting the enterprise and ultimately the client. That 

20 is, the client's options within window 133 are pre-selected and preferred by the 
hosting enterprise based in part on data about the client, details about the client's 
communication equipment and software, and enterprise rules and constraints. In some 
embodiments, the client may aid in customizing window 1 33. However, as it is 
desired by the enteiprise to provide service in a cost-effective manner, the client will 

25 be presented with options as preferred by the enterprise in most cases. 

To further illustrate, refer now to new client section 135. If window 1 33 is 
part of the enterprise WEB page, as is the case with this example, there will be a 
variety of visitors which may or may not be pre-qualified by the enterprise. 
Therefore, an interested party would begin (and be restricted to) taking a new client 

30 survey, illustrated as one of the options in section 135. If the enterprise rules require 
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this as a first step, then the other options may be enabled only upon completion of the 
survey. By choosing new client survey, a second window may contain various survey 
options such as via e-mail, interactive voice recording, type and send method, or the 
like. 

5 Information taken in the client survey is recorded and entered into a C1NOS 

database such as DB 75 of Fig. 1 . Such information may also be compared against 
enterprise rules or constraints, and other known information as may be available to the 
enterprise. Assuming the client is now recognized by the enterprise, the client's media 
hardware and telephony information may be recorded for future interaction purposes. 

10 Such information may include the client's personal computer parameters including 
modem type. Internet connection type, computer platform type, type of Internet phone 
application installed, etc. Similarly, COST telephone parameters may be recorded, 
such as personal phone number, business phone number, cellular phone number, 
forwarding numbers, and so on. Such data will influence latter customization of his 

15 personal window 133 for the particular client including the types of media that will be 
offered. 

Finally, the client may be asked to create a password for the purpose of 
accessing C1NOS. A section 141 is provided containing a network log-in option 
along with download sections for obtaining permanent and or temporary software as 

20 may be desired or needed, or, in some cases, required for the client to access certain 
services, view certain content, and so on. 

Section 137 presents media options for clients seeking customer service from 
the enterprise. These options are, in a preferred embodiment, presented in a 
customized or personalized fashion within the client's window 133 as was described 

25 above. Therefore, each client patronizing the enterprise may access a version of 
window 133 that differs in look and functionality than that of another client. In this 
example, service section 137 contains options for e-mail, chat program, fax program, 
a self-help wizard, and a voice wizard. Other media types may be added or 
subtracted from the client's window 133 depending on any of several criteria. 

30 Personalization of widow 133 takes into account client information as stored in 
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CINOS database 75, service-agent media availability and preferences, and perhaps 
any overriding enterprise rules. Unless and until a client is identified there are 
typically no options presented to the client for continuing a transaction with the 
enterprise . 

5 For an identified client, by selecting the e-mail option, the client's preferred e- _ 

mail program may be activated for the purpose of sending a message to or soliciting a 
reply from a service agent. By selecting chat program, the client may be launched 
into a scheduled service seminar featuring many clients interacting with a service 
expert regarding a certain subject. One enterprise rule regarding section 137 may be 

10 that there is no telephone or 1-phone media option for customer service for a client in 
the absence of an ongoing project with the particular customer. In this sense an 
ongoing project includes any unfinished business that the client is involved in with the 
enterprise. 

Self-help wizards and voice wizards as illustrated in section 137 may be 
15 offered to help a client resolve an issue without taxing further resource. Such wizards 
may be customized based on a client's recorded data, perhaps confirming past 
interactions, providing account or order status, and so on. In some embodiments, 
selecting an option might avail several additional options. For example, selecting chat 
program may avail three possible chat programs to choose from with different 
20 schedules, content, and functionality attributed to each individual program. 

New order section 139 in this example contains various options adapted to 
facilitate placing orders. The options as illustrated herein include, but are not limited 
to. I-phone, call back, promotional models, video presentations, an on-line viewer, 
and an order wizard. Interaction is the same as was stated with regards to section 137. 
25 For example, selecting promotional models, accesses a database containing the current 
promotional information and features of products which may be viewed interactively 
by the client using an on-line viewer offered as one of the functional options (tool). 
The options presented in the New Orders section may also be customized according to 
client identity, demographics, transaction history, and enterprise rules. 
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On-line viewers may enable the client to view documents that are not 
supported on his computer platform. Selecting video presentation may avail several 
types of videos for viewing ,such that the client may choose one. If the client does not 
have a viewer installed on his computer which will support the offered video, perhaps 
5 the on-line viewer may play the video, or the client could download a temporary 
viewer from section 141 , etc. Selecting call back may bring up a second array of 
media choices made available by the enterprise for receiving a reply interaction from 
an agent. 

By providing a controlled interface window such as window 133 the enterprise 
10 may control routing and interaction right from the beginning of customer contact. 
Through the innovative method of linking and reporting to other C1NOS functions, 
and repositories, much real-time personaliation of window 133 according to enterprise 
rules and customer parameters may be made automatically. For example, if a client's 
history indicates a propensity toward frequent buying, an 1-phone option may be 
] 5 presented in customer service section 1 37 in his window 1 33 immediately after such a 
determination so that he may get direct customer service at all times. 

Certain media options, as described above, may be afforded a certain priority 
over one another regarding interaction with the enterprise. For example, a VIP client 
may have live interactive media choices offered in window 133 such as 1-phone, call 
20 back to COST phone, video phone, etc. A client known for infrequent contact or 

troublesome interactive history may be limited to text-based interaction such as e-mail 
and so on. 

As an integral part of C1NOS functionality, window 133 acts as a portal 
through which existing and potential clients may be screened, categorized and routed 

25 according to enterprise rules. Customer interfaces such as window 1 33 may be 

provided at various locations on a WAN such as the Internet without departing from 
the spirit and scope of the present invention. Such portals may exist in different 
geographic regions, and may be created for differing customer bases such as one for 
Latin America, and one for the pacific rim, and so on. Instances of CINOS routine 

30 may be distributed widely over a network. 
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Although the example provided herein is of a WEB form, it will be apparent to 
one with skill in the art that a CT1 counterpart may be created for the COST telephony 
network. Such a case may be a CINOS enhanced IVR at an SCP or customer access 
point in the COST network. 

5 CINOS, as previously described, optimizes customer/agent interaction in a 

manner which is economical and cost efficient to both the enterprise and the 
patronizing client. The customer interfacing window as taught herein with regards to 
Fig. 5 is innovative in that it is a fully customizable portal that facilities seamless 
integration between clients and enterprise agents according to enterprise rules. 

10 Further innovation is evident in that client data is fully and seamlessly integrated with 
CINOS intelligence and enterprise rules regarding routing of interactions and other 
constraints or limitations that are programmed into the system. In effect, logic from 
the front end. or customer side, to the back end or agent side is linked and accessible 
to all appropriate CINOS routines which include applicable CT1 CINOS routines. 

15 The various customer interfacing logic is are explained more fully below in a series of 
process logic steps in a flow chart. 

Fig. 6 is a flow chart illustrating media-presentation and customer-interface 
logic steps according to an embodiment of the present invention. In step 145, a visitor 
registers at an enterprise's WEB page. The visitor is identified according to enterprise 

20 rules in step 1 47. In step 148 CINOS determines the current status of the visitor after 
searching known client and contact data records. For example, the visitor may be a 
potential new client, an existing client, or an existing business contact. Although not 
specifically illustrated, a potential or new-business-client is not typically logically 
separated from a potential new-client until further process ensues in step 149 with 

25 regards to qualification via survey. 

If the visitor wishes to be a client, he may log-in to the network system in step 
1 59. Log-in may be automatic in the event that CINOS remembers the client's 
assigned password, or perhaps typing the password or other code may still be required 
for security reasons. At the time of log-in. window 133 is presented in personalized 

30 fashion according to client data and enterprise rules in step 161. In step 1 63, 
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interaction between an enterprise entity and the client begins with a media type that is 
offered by the enterprise and selected by the client. An enterprise entity, as 
immediately described above, is herein defined as an agent, knowledge worker, 
service person, or any other live attendant, as well as any entity constituting an 
5 automated response action such as an automated fax, an 1 VR, automated file 
downloads, etc. 

At step 148, if it is determined that the visitor is new. then a new client survey 
is conducted in step 149. Step 149 will determine if the new visitor is a client or 
business contact via the survey process. As described with reference to Fig. 6. the 

10 client survey may be conducted using a variety of known techniques and media. 

Presuming that a new visitor qualifies as a client or business contact in step 149, he or 
she may be asked to create a password in order to provide access to C1NOS. In step 
1 53, the client's appropriate communication and system parameters are recorded for 
future reference and for use in customizing window 133. 

15 At step 155, a client instance of C1NOS, or perhaps another enabling 

application, may be presented for download by the client. In some embodiments, 
there may be no required software for download. Therefore, step 1 55 may be 
considered optional in this regard. In step 1 57, the new client may log-in to the 
network system and begin interaction. Because the client, in this case, is accessing 

20 the system for the first time, the steps wherein he would obtain a customized window 
and begin interaction with an enterprise entity are not shown as intermediate 
configuration of media choices, product preferences, and the like, may still be 
required before a customized interface may be presented. In one embodiment, the 
client may not see a customized window until the next time he or she attempts to 

25 access the network. 

Steps 165, 167, and 169 for an existing business contact as determined in step 
148 are similar to steps 159, 161. and 163 for an existing client although the rules for 
interaction such as media used, personnel involved and so forth will be different. For 
example, in step 167 an existing business contact may be offered the option of using a 
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network-collaboration program wherein I-phone, file sharing, video conferencing and 

the like are inherent to that one application. 

It will he appreciated that there are many possible logic sequences or steps that 

may be followed in interfacing and enabling interaction between a client and an 
5 enterprise entity without departing from the spirit and scope of the present invention. 

Fig. 6 presents just one possible example of many. 

It will be apparent to one with skill in the art that the rules governing the types 

of media offered to clients may be based on a combination of variables such as may 

be decided upon by the enterprise without departing from the spirit and scope of the 
10 present invention. Likewise, offered media types may be added or withheld from a 

client over a period of time based on such variables. Moreover, such additions or 

subtractions of media availability with regards to customer interface window 133 may 

be automated and based on calculated variables. 

In one embodiment, a client may add or subtract media choices if desired, 
] 5 however, the enterprise may reserve the right not to engage such media if added by a 

client. 

In one embodiment, special application-independent media viewers such as the 
viewer offered in section 139 of window 133 of Fig. 5. are offered to clients and 
possessed by agents so that initial illegible information may be made human readable 
20 regardless of the authoring application used by the agent or the client in the process of 
interaction. 

Rules-Based Storaee and Threading of Multimedia Interactions 

25 In a preferred embodiment of the present invention, all CINOS controlled 

interactions with customers or business contacts are recorded and stored in a contact 
history comprising a MIS database and a text database such as were described with 
reference to copending application P331 3PA. and described above. That is, actual 
multimedia interactions are recorded to one database or to a section of one database 

30 supporting all multimedia types used in the communication center, and text-based 
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versions are stored in another database or portion of the same database. All of the 
actual recorded transactions and text versions are related as a threaded contact history 
which may be separate from or part of the same database as will be further explained 
below. 

5 Fig. 7 is an exemplary overview of a multimedia-interaction storage system 

within a communication center according to an embodiment of the present invention. 
A system architecture 171 is illustrated solely for the purpose of providing just one of 
many possible architectures in which the methods and apparatus of the invention may 
be practiced. Architecture 171, which in a preferred embodiment comprises both 

10 conventional and DNT apparatus, is exemplary of an architecture that could facilitate 
C1NOS according to an embodiment of the present invention such as is also the case 
of Fig. 1 

At the heart of the storage system is a mass-storage repository 1 87 adapted to 
store multimedia interactions as well as text-based related files. Repository 1 87 may 

15 utilize any form of digital storage technology known in the art such as Raid- Array, 
Optical Storage, and so on. The storage capacity of repository 187 will depend 
directly on it's implementation with regard to the size of the communication center 
and predicted amount of data that will be stored and kept by the system. 

In this example, repository 1 87 is divided logically into two sections. One 

20 section, multimedia information system (MIS) 1 89, is responsible for housing all 
multimedia interactions defined as media that is not text-based such as audio, video, 
and graphics-based media. All multimedia interactions are stored in MIS 1 89 whether 
incoming, outgoing, or internal. A second section, herein referred to as text section 
191 is responsible for all text-based interactions as well as text versions related to 

25 non-text files. Sections 191 and 1 89 of repository 1 87 are analogous to MIS 79 and 
DB 75 of Fig. 3. 

Repository 187 is connected to a communication-center local area network 
(LAN) 195. Repository 187 is accessible via LAN 195 to authorized personnel within 
a communication center such as agents, knowledge workers, or the like, and may, in 
30 some instances, also be made available to clients communicating with the call center. 
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A network router (RTN) 1 75 is shown connected to LAN 1 95 via network connection 
203. In this example, network router 175 is the first point within a communication 
center wherein DNT media arrives. Network router 175 is exemplary of many types 
of routers that may be used to route data over LAN 195. An Internet-protocol- 

5 network-telephony (IPNT) switch 1 76 is connected to network router 175 via data link 
as is known in the art. IPNT switch 1 76 further routes or distributes live IPNT calls 
that do not require routing to a live agent. IPNT calls that are routed to live agents are 
sent over connection 203 to LAN 195 where they reach agent PC/VDLPs (not shown) 
or DNT- capable phones (not shown) as illustrated via directional arrows. 

10 An object of the present invention is to record all multimedia interactions and 

store them in MIS 1 89. A further object of the present invention is to similarly record 
text versions of and text files related to all multimedia interactions and to store them 
in text-based section 191. For the purpose of the present invention, a text version of a 
non-text file is defined as a sufficient text rendition or description of a corresponding 

1 5 multimedia interaction. Still another object of the present invention is to provide an 
innovative mechanism wherein authorized persons may access any particular block of 
text and if desired, call up the actual media to which the text relates. 
More detail regarding the order and manipulation of repository 1 87 is described 
further below. 

20 Creating text-based versions of live multimedia interactions may, in some 

cases, be accomplished via an automated method. For example, a digital voice 
attendant 197 is provided and linked to IPNT switch 176. Digital voice attendant 197 
may be of the form of a DNT-capable I VR or other digital voice-response mechanism 
as may be known in the art. Such automated attendants may interact with a voice 

25 caller instead of requiring a live agent. A speech to text converter 199 is provided and 
linked to voice attendant 197. As digital voice attendant 197 interacts with a caller, 
speech to text converter 199 uses voice recognition technology to convert the audio 
speech to text. Such text may then be stored automatically into text section 191 and 
related to the also-recorded audio data. 
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lt will be apparent to one with skill in the art that as speech recognition 
technologies are further improved over their current state, which is adequate for many 
implementations, reliable text versions of audio transactions are not only possible but 
practical. Such speech to text conversions are used here only for the convenience of 
5 automation wherein no live attendant is needed to transcribe such audio data. The 
inventor is familiar with such converters as used in the C1NOS system according to a 
preferred embodiment. Such converters provide convenience in the practice of the 
present invention but are not specifically required to achieve the objects of the present 
invention. 

10 Manual transcription may also be used to convert audio/video to text or code 

that may then be entered into text section 191. For example, a live attendant 201 is 
shown connected to LAN 1 95. Attendant 201 , in this case, may be given the 
responsibility of transcribing audio files from speech to text and annotating video or 
graphics files for the purpose of creating text files related to the non-text data. One or 

15 more live attendants such as attendant 201 may be provided for this purpose. Some 
media arriving at a communication center such as the one represented via architecture 
171 will already be text-based and therefore require no conversion or annotation. 
Short e-mails, Faxes, word documents, and so on are part of this media category. 

An automated services system 193 is illustrated as having a direct connection 

20 to section 1 91 of the data repository. System 1 93 is provided for certain text-based 
interactions, as described above, wherein a complete text record of the interaction may 
be mirrored, or otherwise created and stored into text section 191. Such automated 
services may include but are not limited to automated e-mail and fax systems. For 
example, a fax may be sent and mirrored into section 191 or, perhaps recreated using 

25 an optical character recognition (OCR) technique and then entered. Physical text- 
documents such as legal papers and the like, may be automatically scanned into text 
section 191 before they are sent to clients. There are many possible automated 
techniques for entering text files into a database. Such methods described with 
regards to automated services 1 93 are a convenience in practicing the present 
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invention but are not specifically required to achieve the objects of the present 
invention. 

With respect to the dual capability (COST/DNT) of architecture 171. a central 
telephony switch 173 is provided to be a first destination for COST calls arriving 

5 from, for example, a PSTN network. Switch 1 73 may be a PBX, ACD, or another 
known type of telephony switch. Internal COST wiring 182 connects telephony- 
switch 1 73 to agent's individual telephones (not shown). Switch 1 73 is enhanced via 
a processor 179 running an instance of T-server and an instance of Stat-servcr ; which 
are software enhancements known to the inventor and have been previously described. 

10 Such enhancements provide CT1 applications, such as intelligent routing, but are not 
specifically required to practice the present invention. C1NOS as previously described 
is adapted to be integrated with such software when present in a C1NOS enhanced 
communication-center. 

An intelligent peripheral in the form of a COST 1VR 177 is provided for the 

1 5 purpose of interacting with callers seeking information and the like who do not 
require connection to a live agent. 1VR technology may comprise voice response, 
touch tone interaction, or a combination of these technologies. 1VR 177 is linked to 
processor 179 and also to automated services 193. An example of an IVR interaction 
may be the presentation to a caller of options for using an automated service such as 

20 those described above, or perhaps waiting for a live agent. 

A CTI to DNT interface 1 81 is provided for the purpose of converting COST 
interactions to digital mode compatible with DNT so as to be adapted for digital 
storage and interaction according to CINOS functionality and enterprise business rules 
as described above. Interface 181 is not specifically required to practice the present 

25 invention so long as appropriate application programming interfaces (API's) are 

provided for equipment that interfaces with CINOS. Bi-directional arrows illustrated 
between interface 181 and IVR 177 represent the ability to route interactions in either 
direction. COST to DNT conversion may be accomplished in IVR 1 77 in addition to 
or in place of interface 181. The connection architecture presented herein is 

30 exemplary only. 
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A speech to text converter 1 85 is provided for converting audio from the CT1 
side to text for entering into text section 1 91 as was taught with regards to converter 
199 on the DNT side. Actual recorded media interactions are illustrated entering MIS 
1 89 after text versions are rendered and entered into section 1 9L however this is not 
5 required. In some instances text versions of multimedia interactions may be rendered _ 
after the interaction is stored. There is no limitation regarding sequence. It is 
sufficient to say that converters 185 and 199 are capable of real-time conversion and 
entry. 

Server 1 83 shown connected to LAN 1 95 is adapted to host a C1NOS 

10 MGR.(operating system) application which provides control and organization with 
regard to various functions provided by the C1NOS system as a whole. The storage 
architecture represented herein by element 17K and all it encompasses in this 
embodiment, is meant only to be an example architecture as may be dedicated to the 
storage and organization of communication-center data according to enterprise rules. 

15 A unique method termed multimedia threading by the inventor is used in a 

preferred embodiment of the present invention for relating each multimedia 
interaction whether incoming to, out going from, or internal to the system, such as 
between an agent and a supervisor. This innovative process allows agents or other 
authorized personnel to access text data and ability cross-reference the data to actual 

20 recorded multimedia interactions which may be displayed and played back. 

Fig. 8 is an illustration of a relational diagram as might be displayed on a 
display monitor, representing entities stored in the databases described. The blocks of 
Fig. 8 illustrate threaded text-blocks and their relationship to stored multimedia 
according to an embodiment of the present invention. Repository 1 87 comprises 

25 section 191 and 189 as illustrated in Fig. 7. Section 191 contains text versions of 
interactions that are related by such as chronology, issue, participants, company 
affiliation, and the like. The text documents and versions of non-text files, 
represented in this case by icons, are shown related by serial position. For the sake of 
clarity regarding the innovative threading according to an embodiment of the present 

30 invention, a brief description of prior art threading follows. 
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Threaded dialog as is known in prior art involves a system of strings or threads 
that are identified as being inherent to a single entity or subject matter wherein the 
dialog (questions and replies) is about that subject or about a question or subject that 
an entity has brought forth. A threaded dialog may be finite dialog (is closed at some 

5 point) or it may be ongoing. Typically, a thread, which connects or associates the 
pieces of dialog, contains all of the dialog in the order that it happened such as in 
chronological order. Threading may be implemented based on other criteria as may 
be appropriate for a particular situation or by particular rules. 

As previously described with reference to the background section, prior art 

10 threading techniques are confined to text such as with an on-line message board or the 
like. The inventor knows of no system that supports full multimedia interaction. The 
innovative implementation taught below integrates the text-based thread with stored 
multimedia interactions such that one may interact with the thread and access various 
stored media associated with the thread. 

15 Referring again to Fig. 8. a customer 205 is illustrated as having two threads. 

They are issue 1 and issue II. Customer 205 has an assigned number XX-XX that 
identifies him or her with respect to the C1NOS system. Issues I and II may comprise 
sales dialog, purchasing dialog, or any other type or purposed dialog as may be 
generic to the hosting enterprise. Customer 205 may well be a business contact, or 

20 even an internal agent practicing dialog with a supervisor or the like. 

A series of icons 225-239 represent the type of media stored for each text 
block (text not shown). For example, issue 1 comprises first an e-mail text followed 
by a fax text, WEB text, and V-phone text. In this case, a time stamp or other known 
method may be used to insure that each text block is in order. Icons 225-239 are 

25 interactive pointers or links to the actual media interactions that they represent. That 
is, the first block of e-mail text is associated with an interactive icon, in this case icon 
225. By clicking on the icon a with a pointer device, the actual e-mail may be 
accessed and viewed. In an alternative aspect, not only the actual transaction may be 
presented to a user for review, but related files may also be listed or otherwise 

30 presented for selection and review. 



WO 00/16207 



PCT/US99/20461 



-37- 

A logical link 207 represents cross-referencing capability between sections 

191 and 1 89. Dialog may be threaded according to a wide variety of business rules. 

For example, a thread or string may represent dialog about a customer, product, agent, 

group of agents, group of customers, and so on. An identifier is assigned to an entity 
5 and to all the communication events to or from that entity, or those in which the entity 

may have been involved such as a group discussion or chat. In this way all 

interactions may be organized and stored accordingly. 

In one embodiment, keyboard commands could be used to cross-reference to 

actual media instead of icons. In another embodiment, text versions of actual media 
10 arc fully viewable with the text itself appearing in interactive form whereupon a 

double click may call up the associated media and so on. There are many variations 

within the scope of the invention. 

Although actual recorded media interactions are. in this embodiment, stored in 

MIS 1 89, there does not have to be two separate databases (one for text and one for 
15 actual media). All data may reside in one database and be sectioned in storage. For 

example, one click on the customer name may bring up text only, while a double click 

on the text brings up the associated media. 

In MIS 1 89, recorded multimedia interactions are represented by icons 241- 

247 and 25 1 . For example element 241 represents all recorded Video phone 
20 interactions. Element 243 represents all e-mails. Element 245 represents all recorded 

COST interactions. Similar associations are made with respect to elements 247 and 

251 which represent WEB interactions and Video mails respectively. WEB 

interactions 247 may include on-line orders, requests, information forms, signed 

certificates, and so on. 
25 Element 249 represents additional stored multimedia files dedicated to, for 

example, promoting the enterprises products or services. Promotional files 249 may 

comprise files of the form of any enterprise supported multimedia. These files may be 

tools that can be sent to clients upon request or perhaps periodically. 

Referring again to section 191. element 227 located on the thread labeled issue 
30 I represents text from a fax. Because a fax is text-based and not a multimedia 
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interaction, there is no corresponding media event associated with it. However, the 
fax is threaded into the dialog according to ; in this case, chronological order. A short 
example of a proposed dialog concerning issue I follows. 

Element 225 represents an e-mail sent by customer 205 to the enterprise 

5 requesting pricing information. An enterprise agent responds with a fax 227 to 

customer 205 containing the requested information. Customer 205 then places an on- 
line order 229 along with a request for confirmation via video phone 23 1 . Issue J may 
be closed at this point. Issue II may represent a threaded dialog concerning company 
service with regards to the customer order of issue 1, or perhaps an agent-to- 

10 manufacturer dialog regarding how the order was handled with respect to issue I. 

In accordance with CINOS functionality as previously taught in descriptions 
above, data may be mined from repository 1 87 for the purpose of enhancing service to 
customer 205. Mined data may be used to affect routing of interactions, product 
promotions or advertisements that may be sent to customer 205. In some cases, 

\5 mined data may effect new dialog with a customer or business contact resulting in 
new thread additions. A complete contact history with interactive linking to actual 
recorded media enables the enterprise to resolve disputes more easily, better service 
the customer, and enhance profitability for the enterprise. 

Fig. 9 is a process flow chart illustrating logical steps taken when building a 

20 threaded multimedia contact-history of communication-center interactions according 
to an embodiment of the present invention. Logical process steps as illustrated herein 
are meant to represent just one of many sequences which may be implemented when 
building a multimedia-threaded contact-history. Actual steps will depend on 
enterprise rules. In step 209, a current interaction to be recorded is identified. 

25 Identifiers may include special passwords or codes for identifying the contacts 

involved with the interaction. The media type of the interaction is identified in step 
211. If the media type is already text-based ; as confirmed in step 213, then the 
interaction is prepared for entry into a database such as section 191 of Fig. 8. 
Preparation may include such automated processes as scanning, mirroring, file 

30 conversions,, and so on. Manual annotation via live attendants such as attendant 201 
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of Fig. 7 may also be performed. In step 223. the text interaction is entered into 
section 191 of repository 187 and takes it's place along the associated dialog thread 
according to enterprise rules. 

If the interaction is of the form of non-text media as identified in step 211, 

5 then the MIS section of repository 1 87. or section 1 89, is notified to accept the input. 
At step 219, the non-text interaction is recorded into section 189 of repository 187. 
This may occur in real time as the interaction takes place, or some point after the 
media interaction was recorded. 

In step 221 , a text version of the recorded media or a text-based document 

10 related to the transaction is rendered for storage into section 191 as part of the thread. 
In some instances, as described with reference to Fig. 7. step 221 is automated via 
speech to text converters and occurs at the same time or before the recorded 
multimedia interaction is entered into section 1 89. In other instances, text versions of 
multimedia interactions may be rendered after the recorded interaction is stored. A 

15 live attendant such as attendant 201 of Fig. 7 may be assigned to parse video and or 
audio for applicable text. Such parsed text is entered into section 191 and takes it's 
place along the thread as was described above. 

In all cases, an identifying medium is used to assign portions of an ongoing 
dialog to the proper location along a thread as well as provide identification to actual 

20 recorded media for cross-referencing such as may occur during a system audit or 

contact review. Further, the appropriate icons and or links are created and associated 
to entered text wherein actual multimedia may be cross-referenced in interactive 
fashion. Hence, the type of media may be readily identified by an auditing or 
reviewing agent simply by browsing the threaded text with accessibility to the 

25 recorded events made by interactive method such as clicking an icon with a pointer 
device as was previously described. As an additional benefit all of the threaded 
dialog, whether text based or not, is rendered in a form that data mining may be used 
to create many useful relationships and to derive much useful information from the 
stored data. 
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It will be apparent to one with skill in the art that the order and specific 
function of logical steps as taught herein may vary according to the type of enterprise, 
existing enterprise rules, and so on. For example, instead of threaded dialogs being 
inherent to a specific customer with the dialog being about the customers interactions, 
5 it may be specific to a particular agent with the dialog about the agents activities. 
Such differences in thread assignment may be incorporated into one rules-based 
repository. 

It will be apparent to one with skill in the art that CINOS may be implemented 
in a single communication center, or in a plurality of communication centers linked 

10 via WAN without departing from the spirit and scope of the present invention. 

It will also be apparent to one with skill in the art that rules may be created 
which govern access to CINOS without departing from the spirit and scope of the 
present invention. For example, customers may be required to subscribe to CINOS. 
and may also be provided with a customer application enabling such access. In 

LS another embodiment, access may be given to the general public according to 
established security rules governing commerce, financial transactions, and other 
processes. 

There are many existing and future implementation opportunities for an 
interaction operating system such as CINOS many of which have already been stated. 
20 The spirit and scope of the present invention is limited only by the claim that follow. 
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What is claimed is: 

1 . A system for relating multimedia events in a multimedia-capable call center, and 
5 making the events available to users, comprising; 

a data repository adapted to store files recording both text-based and non-text- 
bascd events involving the call center; 

recording systems for recording events of all text-based and non-text events 
experienced at the call center; 
10 a system for relating the stored files in one or more serial strings according to 

relational criteria; and 

an interactive display interface for displaying identifiers of the stored files, the 
identifiers arranged in a serial string according to the relational criteria; 

wherein selected events involving the call center are recorded, and a user, be 
15 selecting identifiers, may play individual ones of the stored files, thereby reviewing 
the recorded events. 

2. The system of claim 1 wherein the identifiers comprise icons displayed on a 
computer video monitor, the icons arranged in a serial string. 

20 

3. The system of claim I further comprising a system for preparing text versions of 
non-text events, and for storing the text versions in the data repository related to the 
non-text versions in a manner that the text versions may be accessed by the interactive 
display interface. 

25 

4. The system of claim 3 further comprising a data extraction facility for deriving 
information from text files, including the text versions of non-text files. 

5. The system of claim 4 further comprising a facility for relating informational files 
30 to text files, and therefore to non-text files associated with text files, wherein the 
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informational files may also be accessed by the interactive display interface along 
with the recorded events and text versions of recorded events. 

6. The system of claim 1 wherein the relational criteria include one or more of 
5 chronology, participants in an event : organizational affiliation, and single issue 

relationship. 

7. The system of claim 1 wherein all communication events involving the call center 
are copied, related and stored. 

10 

8. The system of claim 7 wherein a text version of every non-text event is prepared, 
related and stored. 

9. The system of claim 1 wherein events include one or more of e-mails, video-mails, 
15 faxes, file share, file transfers, video calls, chat sessions, DNT calls, COST calls, and 

voice mails. 

10. A method for relating multimedia events in a multimedia-capable call center, and 
making the events available to users, comprising steps of; 

20 (a) storing both text-based and non-text-based events involving the call center 

in a data repository adapted to store multimedia files; 

(b) relating the stored files in one or more serial strings according to relational 
criteria; and 

(c) providing an interactive display interface on a computer video monitor, the 
25 interface adapted for displaying identifiers of the stored files, the identifiers arranged 

in a serial string according to the relational criteria, wherein user, be selecting 
identifiers, may play individual ones of the stored files, thereby reviewing the 
recorded events. 
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1 1 . The method of claim 10 wherein in step (c) the identifiers comprise icons 
displayed on the computer video monitor, the icons arranged in a serial string. 

1 2. The method of claim 1 0 further comprising a step for preparing text versions of 
5 non-text events, and for storing the text versions in the data repository related to the 

non-text versions in a manner that the text versions may be accessed by the interactive 
display interface. 

1 3. The method of claim 12 further comprising a step for extracting information from 
10 text files, including the text versions of non-text files. 

14. The method of claim 13 further comprising a step for relating informational files 
to text files, and therefore to non-text files associated with text files, wherein the 
informational files may also be accessed by the interactive display interface along 

15 with the recorded events and text versions of recorded events. 

1 5. The method of claim 10 wherein the relational criteria include one or more of 
chronology, participants in an event, organizational affiliation, and single issue 
relationship. 

20 

1 6. The method of claim 1 0 wherein, in step (a) all communication events involving 
the call center are copied, related and stored. 

1 7. The method of claim 16 wherein a text version of every non-text event is 
25 prepared, related and stored. 

1 8. The method of claim 10 wherein events include one or more of e-mails, video- 
mails, faxes, file share, file transfers, video calls, chat sessions, DNT calls, COST 
calls, and voice mails. 



WO 00/16207 



PCT/US99/20461 




WO 00/16207 



2/9 



PCT/US99/20461 




WO 00/16207 



PCT/US99/20461 



3/9 



103 



105 



107 



109 



Incoming Event 



101- 



Consult Knowledge 
Base 



Yes 



Route to Best Fit 



Record Entire 
Transaction 



Prepare Maximal 
Text-Based Version 



Extract relevant 
data from text 
version and store in 
knowledge base 



Customer and 
Media Type 
Identified 



Fig. 3 



97 




113 



115 



111 



Record Entire 
Transaction 



Extract relevant 
data from text 
version and store in 
knowledge base 



Route based on 
extracted data and 
data in knowledge 
base 



WO 00/16207 



PCT/US99/20461 



117 



4/9 



115- 



Agent's Desktop Interface 



119 



12] 



HTML Client 



HTTP 



Internet Information 
Server 



129 



COM 



123 



Windows Client 



Active X 



-125 



Desktop Toolkit 



COM 



Active X 
< 



Business Objects 



131 



COM 



Physical Database 



ODBC 



127 



Existing 
Applications 

CIS 
ERP 

Commerce 
Etc. 



Fig. 4 



WO 00/16207 



PCT/US99/20461 



5/9 




New Client 



Create Password 



Telephony Options 



1PNT Options 



New Client Survey 



141 




Customer Service 



E-Mail 



Chat Program 



Fax Program 



Self-Help Wizard 



Voice Wizard 



Log-in to Network System 



Temporary Software 



Client Software Download 



New Orders 



I-Phone 



Call Back 



Promotional Models 



Video Presentation 



On-Line Viewer 



Order Wizard 



Web Form Customer Interface 



Fig. 5 



WO 00/16207 



PCTYUS99/20461 



6/9 



Register Visitor 



145 



Visitor identified 
as per enterprise 
rules 



149 



151 



153 



355 



157 



Conduct new client 
survey as per rules 



New 



Create password 
if qualified as 
customer or contact 




Log-in to network 
system 



Business 
contact 



161 



Log-in to network 
system 



Configure basic 
communication 
parameters-client 




Present customized 
options based on 
client data/rules 



163 



Present interactive 
network options 
based on rules 



Interact with client 
according to media 
offered and selected 



Present required 
software as per 
enterprise rules 




Log- in to network 
system begin 
interaction 



Interact with contact 
according to preferred 
network application 



Fig. 6 



WO 00/16207 



PCT/US99/20461 




WO 00/16207 



PCT/US99/20461 



9/9 



209 




Record 
Multimedia 
Into MIS 



Text version or 
related doc 
prepared 



223 



Enter text 
into database 



217 



219 




187 



Repository 



Fig. 9 



INTERNATIONAL SEARCH REPORT 



International application No. 
PCT/US99/20461 



A. CLASSIFICATION OF SUBJECT MATTER 

1PC(6) :G06F 15/00, 13/00 

US CL :709/223, 203, 212. 216, 226, 265 
According to International Patent Clas sification (IPC) or to both national classification and IPC 

R FIELDS SEARCHED 

Minimum documentation searched (classification system followed by classification symbols) 
U.S. : 709/223,203,212.216.226.265 



Documentation searched other than minimum documentation lo the extent that such documents are included in the fields searched 



Electronic data base consulted during the international search (name of data base and, where practicable, search terms used) 



C. DOCUMENTS CONSIDERED TO BE RELEVANT 



Category* 


Citation of document, with indication, where appropriate, of the relevant passages 


Relevant to claim No. 


NONE 


NONE 




NONE 


Y 


US 5,181,239 A (JOLISSAINT) 19 JANUARY 1993, COL. 2, 
LINES 31-32, COL. 3, LINES 37-42, COL. 2, LINES 19-26, COL. 
2, LINES 47-53, COL. 4, LINES 60-COL. 5 LINE 10, COL. 5, 
LINES 11-59. 


1-18 


Y,P 


US 5,867,494 A (KR1SHNASWAMY ET. AL.) 02 FEBRUARY 
1999, COL. 79, LINES 26-34, COL. 24, LINES 54-58, COL. 69, 
LINES 24-49, COL. 27, LINES 54-58, COL. 32, LINES 18-19, 
COL. 68, LINES 9-22, COL. 28, LINES 7-9, COL. 28, LINES 46- 
51, COL. 37, LINES 39-55, COL. 38, LINES 44-47, COL. 2, 
LINES 46-51, COL. 145, LINES 1-40, COL. 8, LINES 9-22, COL. 
183, LINES 19-30, COL. 36, LINES 24-32, COL. 42, LINES 24- 
35, COL. 42, LINES 41-44. 


1-18 


|"x| Further documents are listed in the continuation of Box C. See patent family annex. 


Special categories of cited document*. 

'A' document defining the general state of the art which a ni>t considered 
to be of particular relevance 

•E* earlier document publuhcd on or after the international Tiling dale 

"1/ document which may throw doubu on priority claim(») or which ts 
cited to e-slabhsh the publication date of another citation or other 
special reason fas specified! 

•O* document r of erring to an oral disclosure, use. exhibition or other 
meant 

"P* document publuhcd prior to the international filing date but later than 
the priority date claimed 


*T" later document published after the international filing date or priority 
date and not in conflict with the application but cited to understand 
the principle or theory underlying the invention 

•X" document of particular relevance; the claimed invention cajuiot be 
comideied novel or cannot be considered to involve an inventive step 
when the document 11 taken alone 

*Y* document of particular relevance; the claimed invention cannot be 
considered lo involve an inventive step when the document is 
combined with one or more other such documents, such combination 
being obvious to a person skilled tn the art 

"Jt* document member of the same patent family 


Date of the actual completion of the international search 
10 DECEMBER 1999 


Dale of mailing of the international search report 

23 DEC 1999 


Name and mailing address of the ISA/US 
Commissioner of Patents and Trademarks 
Box PCT 

Washington. D.C. 20231 
Facsimile No. (703) 305-3230 


Authorized officer 

LE H1EN LUU Q^rrUL& &. iT^^ht^ 
Telephone No. (703) 305-9650 



Form PCT/ISA/210 (second sheet)(July 1992)* 



INTERNATIONAL SEARCH REPORT 



International application No. 
PCT/US99/20461 



C (Continuation). DOCUMENTS CONSIDERED TO BE RELEVANT 


Category* 


Citation of document, with indication, where appropriate, of the relevant passages 


Relevant to claim No. 


Y 


US 5,715,307 A (ZAZZERA) 03 FEBRUARY 1998, COL. LINES 
32-40, COL. 3 LINES 65-COL. 4 LINE 20, COL. 1 LINES 18-21, 
COL. LINES 59-64, COL. 2 LINES 31-53, COL. 4 LINES 60- 
COL. 5 LINE 10. 


1-18 



Form PCT/ISA/210 (continuation of second sheet )( July 1992)* 



